The Reliability of PLATO 1 Running Head: THE RELIABILTY OF PLATO Investigating the Reliability of Classroom Observation Protocols: The Case of PLATO

The reliability of the PLATO rubric is defined using Generalizability Theory and FACETS. Results from two G-Theory analyses reveal that the reliabilities of scores from individual PLATO items and the total score across all items are most strongly affected by the number of segments in which teachers’ performance is observed. For example, it is reported that a single rater should observe at least 5 segments in order to achieve a reliability coefficient greater than .80 for a score that represents the sum of scores form the separate items in the rubric on both a 12 and 7-item version of the Rubric. Reliability curves are generated to inform potential PLATO users on how measurement design characteristics affect reliability. Further, FACETS results reveal systematic effects due to occasion of measurement, segment, and rater. Scores adjusted for the specific measurement circumstances of individual observations are reported and discussed. The Reliability of PLATO 3 Investigating the Reliability of Classroom Observation Protocols: The Case of PLATO Two sets of analysis are reported in order to establish the measurement properties of the Protocol for Language Arts Teaching Observation (PLATO) rubric. First, the expected reliability of the PLATO rubric for different measurement schemes is estimated using techniques from Generalizability Theory (See Cronbach, Gleser, Nanda, and Rajaratnam, 1972 or Brennan, 2001). Second, the measurement properties of scale scores calculated from the particular data collected in the present study are determined using FACETS (Linacre, 2010). The results from the G-Theory analysis take into account the effect of error from different raters (judges) and different observation segments on the reliability of the individual item scores as well as scores that represents the composite of the individual items that make up the PLATO rubric. The results from the FACETS analysis take into account the same sources of error as well as error from the occasion of measurement and item sampling to produce scores that are adjusted for these sources of error. While on the surface the two sets of results appear to provide two versions of the same result, they serve different purposes and are based on different assumptions described next. The purpose of the G-theory analysis is to provide researchers interested in adopting the PLATO rubric in their research with guidelines for determining the measurement scheme required to achieve different levels of reliability. The primary finding from the G-theory analysis is a set of reliability estimates that differ depending on the measurement scheme employed. Alternatively, the FACETS analysis specifies the measurement properties of scores that have been adjusted for the specific measurement The Reliability of PLATO 4 circumstances of these particular data. These results are useful for researchers interested in using these specific data in descriptive models. The primary result from the FACETS run is a set of scaled scores and their respective standard errors. Other than differences in the respective uses, it is also important to note how Gtheory and FACETS analyses treat factors in the measurement design differently. In our G-theory analysis all tractable sources of error are considered random effects whereas in any FACETS analysis tractable sources of error are treated as fixed effects. Hence, results from the G-theory analysis become predictive of the reliability expected if for example different numbers of different raters are used to produce the data, whereas results from the FACETS run speak exclusively to the error associated with the particular raters used in the collection of these data for example. We now present a brief description of the PLATO rubric and measurement scheme followed by a presentation of the G-theory and FACETS results respectively. The paper concludes with a brief summary of the results and a discussion of the implications of the findings. The PLATO Rubric and Observation Protocol The following is a very brief overview of the structure of the PLATO rubric. For a more in depth consideration of the theory informing its construction see Grossman et al (2010). The version of the rubric under consideration in the present study consists of 12 elements/items (see Table 1) designed to measure independent aspects of the quality of middle/high school English/Language Arts teacher instructional practices. Table 1 The PLATO Rubric Elements and their Descriptions The Reliability of PLATO 5